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MEASURING SOFTWARE DEVELOPMENT TECHNOLOGY 

BY 


FRANK E. McGARRY 
GODDARD SPACE FLIGHT CENTER 


In late 1976, the Goddard Space Flight Center (GSFC) Initiated effort., to create 
a software laboratory where various softKare development technologler and 
methodologies could be studied, measured and enhanced. This laboratory became 
known as the Software Engineering Laboratory (SEL), and since Its Inception has 
been actively conducting studies and experiments utilizing flight dynamics 
projects in a production environment. The SEL evolved to a full partnership In 
the efforts between GSFC, the University of Maryland and Computer Sciences 
Corporation (CSC). 

The approach that the SEL has taken in carrying out the studies has been *’o 
apply varying methodologies, tools, management concepts, etc. to software 
projects at Goddard; then to closely monitor the entire development cycle so 
that the entire process and product can be compared to similar projects 
utilizing somewhat different approaches. This monitcrlng function led to a need 
to collect, store and Interpret great amounts of data pertaining to all phases 
of the software process, product, environment and problem. This data collection 
and data processing process has been applied to over 40 software projects 
ranging in size from 2,000 lines of code to approximately 120,000 lines of code 
with the typical project running about 55,000 lines of code. 

The data that has been collected (and Is still being collected) and Interpreted 
for these projects comes from 5 sources: 

1. Data Collection forms utilized by programmers, managers and support 
personnel. Typical types of data collected Include: 

o Error and Change Information 
o Weekly Hours and Resources 

o Component Effort (hours expended on each component by week) 
o Project Characteristics 
o Computer Run Analysis 

o Change and Growth History (week by week records of source code) 
(Additional Information is contained In references 1 and 2) 

2. Computer Accounting Information 

3. Personnel Intervlews-during and after the development process 

4. Management and Technical Supervisor Assessments 

5. Tools-used to extract data and measures from source code 
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For tho more than 40 project* which have been monitoredi approximately 21,000 
forms have been processed and are continually used to perform studies of the 
software development process. To support tho storage, validation and usage of 
this information, a data base was designed and built on a PDP-11/70 at Goddard. 
(Reference 3) 

Approach ( Cha r t 2 ) 

The steps that have been taken to carry out the Investigation within the SEL 
have been : 


Develop a profile of the software development process as It Is 
*now'. First we must understand what we do well and what we do not so well so 
we can build a baseline of current characteristics whereby later we can honestly 
measure change . 

2. Experiment with similar type projects. The second step has been to 
apply select tools, methodologies and approaches to software projects so they 
can b*' studied for effect. 

Measure the process and product. As projects are developed which 
are utilizing different software development techniques, the SEL uses the 
extracted data to determine whether or not the applied technology has made any 
measurable impact on the software characteristics (This may include reliability, 
productivity, complexity, etc.). 

Environment (Chart 1) 

The projects which have been monitored and studied are primarily all flight 
dynamics related software systems. This software includes applications to 
sr,: art attitude determination, attitude control, maneuver planning, orbl* 
adjust and general mission analysis. 

The attitude systems normally have ry similar characteristic and all are 
designed to utilize graphics as well as to run in batch mode. Depending on the 
problem characteristics, the typical attitude systems range in size from 30,000 
to over 120,000 lines of code.* The percentage of reused code ranges from less 
than lO percent to nearly 70, percent with the average software package being 
comprised of approximately 30 percent reused code. 

The applications are prir4arily scientific in nature with moderate reliability 
requirements and nor»n* 4 ily are not required to run in real time. The development 
period typically runs for ixbout 2 years (from Requirements Analysis through 
Acceptance Testing). The development computers are typically a group of IBM 
S/360's which have very limited resources and where reliability is quite low 
(typically less than 3 hours MTBF) 

Details describing the environment can be found in Reference 1. 


*Here, a line of code is any 80 byte record processable by a coropiJi*r or 
assembler (l.e., comments are Included) 
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Expcrlroents Completed (Chart 4) 


As was mentioned earlier, the SEL has monitored over 40 software development 
projects during the 6 years of operation. During this time period, numerous 
methologles, models, tools and general software approaches have been applied and 
measured. The summary results to be presented are based on these projects. The 
summary will be divided into 3 topic areas: 

1. Profiles of the Development Process 

2. Models 

3. Methodologies 
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Pro! i U‘s t»! t hi» IMocohh (CharlH S iUrn 1?) 

Tlu‘ liiMt Ktt’p in .It ( ompt I iig to moaiuiro the el iect 1 vtMU'Hs ot any Hottware 
t i‘rhiu> h>>» V is io f’lMUMate a baaeline or profile of how one typically pertorma 
hla )oh« Then .is modlli*Hl approachew are attempted on aimilar projecta, the 
efliU'trt may be .ippaient bv comparlaon, 

KeHOuiees Allt>eatimi (('hart /) 

Om» Mt't o\ b.inii' intormation that oiv may want to underatand la just where do 
puH*iammers api*mi their timi*. When the SKI. looked at n imeroua projecta to 
luultMstaiul wlu'ie the lime was apent , it found that the SKI. environment deviated 
siimewhat t iiim tin* i>ld rule. Typically projecta indicated that when the 

total lu>u! H 1 ‘xpended were based on phase dates of a project (i.e., a specific 
ilata iU*tiiu»il t hi' absolute complet Ion of one phase ot the cycle and the beginning 
ot t hi' next phase> the btisikilown was less than percent tor design, cloae to 
Stl peri'ent lot code and about percent tor integration and teat. 

When tin* pr<)grammers ptovlded weekly data attributing their time to the activity 
that they telt they were actually doing, no matter what phase of software 
development they wt»re in; the pro! i le looks quite different. The 1 phases 
(designs, code, test) each cotisumed approximately the seme percent effort and 
i>vei percent ot t hi» time was attributed to ’other’ activiMes (such as 
travel, tialning, unknt>wn, etc.). The SKI. has continually found that this 
ettort jUMher) t'xists, and cannot easily be reduced, and most probably should be 
accepteii as a given. The SKb has found it to be a mistake to attempt to 
increase productivity merely by eliminating major portions of this ’other’tlme. 

Devoiopinent Resources (Chart 8) 

Another area of concern to the SEL in defining the beeic profile of software 
development, was that of staffing level and resource expenditure profiles. Many 
authorities stibscrlbe to the point that there is an optimal staffing level 
profile which should be followed for all software projects. Such profiles as a 
l^a^leijj^h Curve are suggi'sted as optimal, (’hart 8 depicts characteristics of 
classes ot projects monitored in the SKI. and shows the difference in 
productivity and reliability tor groups of projects having different staffing 
level protiles. Altbotjgb the Rayleigh (Uirve may be acceptable for some 
projects, the SKI, has found that wide variations on these characteristics still 
leaii to a succi'ssful projects. The SKI. has also found that ext re me deviations 
mav he indtcatlvi' oi problem sottware. 

(IV'taileil Intormation can he found in Reference 4 and S) 
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Productivity for large vs, small aystemB (Chart 9) 


The common belief by many software managers and developers Is that as the size 
of a software system Increases, Its complexity increases at a higher rate than 
the lines of code Increase* Because of this fact, It is commonly believed that 
lt« the effort equation 

E - alb 

where E - effort of person time 
where I <■ lines of code 

that the value of b must be greater than 1* The projects that the SEL has 
studied have been unable to verify this belief and Instead have found the value 
of b to approximate *92 In the SEL environment. The fact that this equation is 
nearly linear leads to the counter Intuitive po'.nt that a project of 150,000 
lines of code will cost approximately 3 times as much as a 50,000 lines of code 
project-instead of 4 or 5 times as much as is often commonly believed. 

(Further details can be found In Reference ^.) 

Productivity Variation (Chart 10) 

Another characteristics that the SEL has been interested in studying has been 
the variations in programmer productivity. Obviously one would want to increase 
the productivity by whatever approach found to be effective, but first we must 
clearly understand what the baseline characteristics of productivity are 
(minimum, maximum, average, difference between small and large projects, etc.); 
only then will we know if we have improved or not in the years to come. 

As has been found by other researchers in varying environments, the productivity 
of dlf.;erent programmers can easily differ by a factor of 8 or 10 to 1. The SEL 
did find that there was a greater variation (from very low productivity of .5 
1.0. c/hour to 10.8 l.o.c./hour) in small projects. The probable reason for this 
is that newer people are typically put on smaller projects and the SEL has found 
extreme differences in the relatively Inexperienced personnel. 

Reusing Code (Chart 11) 

As was stated in the introduction, projects being developed In the SEL 
environment typically utilize approximately 30 percent old code. Although It is 
obviously less costly to Integrate existing code into a system rather than 
having to generate new code, there is some cost that must be etributed to 
adopting the old code. The development team must test. Integrate and possibly 
document the old code, so there is some overhead. By looking at approximately 
25 projects ranging in size from 25,000 lines of code to over 100,000 total 
lines of code and ranging in percent of reused code from 0 percent to 70 
percent, the SEL finds that by attributing a value of approximately 20 percent 
overhead cost to reuse code, the expenditures of the 25 projects can best be 
characterized. Now the SEL uses the 20 percent figure for estimating the cost 
of adopting existing code to a new software project. 
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Error Characteristics (Chart 12) 


One of the other characteristics of a software environment that is of great 
concern to developers and managers is that of expected software reliability and 
that of overall software error characteristics. Before attempting to improve 
software reliability or before att'impting to 

minimize the impact that software errors may have, the SEL had to first 
understand the error characteristics of the typical applications software in the 
ftEL environment. 

By collecting detailed error report data and through the monitoring of numerous 
applications projects many error characteristics have been studied. 

Several pieces of information which are depicted in Chart 12 and which are based 
on 1381 error reports from approximately 13 projects include: 

o Most errors are local to one component (subroutine or function) 

o Less than 10 percent of errors were attributed to faulty 
requirements 

o A great percent of errors (48 percent) were estimated to be trivial 
to correct (less than 1 hour) 

o A very low percent of errors (7 percent) were estlmi^ted to be a 
major effort to fix (greater than 3 days) 

(Further statistics and more detailed explanations can be found in References 7 
and 8). 
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Mode 1 8 ( riwi r t 8 M I hi ougli 1 6 ) 

A Hoeoiul set ot 8tiull«*« tliat the SKI, has aetivelv pursued Is that ol evaluatin>^, 
reviewing, and deveh^plng software models. This Inehules resouiee imult^ls, 
reliability models as well as oomplexltv mettles. 

Measiires t oi Si>t t ware U-hart 1 s ) 

The SKI. has attemptei! io ut.llee vat Ions available sottware metrles 1 
eharaotetl/e (lie software pia>iluets gt»nerated. Siu'h mettles as the Mei'abi* 
(!velomatle (\unpu»xltv, llalsteail Keagth, and llttes i>t eode were only a tew ot the 
measutes that wtM e revleweil. 

It Is eoumu>nlv bellevecl that the sl/e ot a et>mponeitt ox the eompl'xltv ot a 
component will b»* eiireetlv correlated to the reliability ot that component . One 
set ot studies pertormed In the SKI. attempted to verity this bi*llet. by taking 
over nSO modules whlcli liad very detailed records oi erriu data, the SKI. computed 
the eoirelatlons ot 4 characteristics ot t lu* compi>nents. The charact er 1 st I c 
Included total lines ot code, executablt* lines ot code, ('vcli>matlc Complexity 
and Halstead Length. The resultaitt cor re lat lints are diplcted in ('hart la; wliich 
shows a very high direct cori«*latlon tor the 4 measures. 

A second study was pertormed wht»re the erroi rate itt eailt ot t Ite component s was 
plotted against sl^^e *is well as against Cyclomatic Complexity. Tht* SKI. expecteil 
ti show (hat larger components have hlghtM* error rates than smallet components 
and t Itat component s ot higher complexity rating had higher error rates. The 
plots on Chart II show that t lu* results were count er - 1 nt ul t 1 ve . Tlie SKI. has 
been unable to verity that larger ot more complex component s Indeed have higher 
ei ror i at es . 

(lost Models (Chart 

In addltloi^ to the studies maile pertaining to various measures tor 
software, the SKI. has also utilized t lie cost data collected from the many 
projects to calibrate and evaluate various available lesource t*st (mat ion models. 
No attempt was intended to quality one model as being any better than another. 
The oblt'ctive ot t lu* studies was to betti*i undi*rstand the Sf ns i t i y i t i es ot the 
various models and to determine which models set»med to characterize the SKI. 
sottware ile ve U>pment environment most cons ( st ent I v . 

In studying these resourct* models, projects which were somewhat similar In 
sire were used as experimental projects. Kach ot the models was ted complete 
and accurate data from the SKI. data base and each was calibrated with nominal 
sets of projects as completely as the experimenters could. Snnmuirv results, 
which are given in Chart IS, Indicate that, occasional ly, some models can 
accurately piedlct effort required tor a software proj«*ct. The SKI. has 
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fi* i 1 1 t'Tvit what iminv othi*i* aottwaro diwo lopora and manaji^erH claim. Coat mocIclH 
should novel ho used as a solo soiiroo o! oatiroatlon, Tho iiaor must have aooosB 
to oxporlonood poraonnol tor oatimatlnp, and 

must also have aoooss to a corporate memory whlcli can he used to calibrate and 
leintorce sonuM>nes estimate ot cost • Resource models can only be used as a 
supplemental tOi)l to reintorce ones estimate or to t lag possible 
Inconsist encles . 

More detailed intonmition on t lie SKL studies can he tound in Reterence I, 10, 
S 

jU'J^ijih i IJ^tjf MiK^e I s ( Oha M I tO 

Another type ot model that t lie SKL has spent some et torts in understanding and 
ca I i hra t l ng t s t he re 1 1 ahl I i t y mode I . Alt hough numerous approaches have been 
suggested as to (ust how one best predicts the level ot error pro.ieness tliat 
sottware may have, t lie SKI. has only pertormed any extended studies on one 
model-that which is attributed to John Musa. The model Is a maximum likelihood 
iiiMhod and t lie SKL attempted to apply detailed tault reports t rom 2 separate 
prolecls to the model In an attempt to determine li the model could accurately 
predict remaining faults In the sottware. 

Chan Indicates that one ot the experiments was (jiilte successtul and one of 
the experiments was not successtul. It should be noted that during and alter 
these experiments, John Musa reviewed the results and the data very carefully 
and lu' has pointed out some possible diM I c I eiic I es In the SKL data which could 
possibly lead lo erroneous results in this application of the reliability model. 
One such piece ot data is the granularity with which computer CPU time Is 
recorded between reported t an 1 1 s . Tlu' SKI. data is not as accurate as the model 
calls t or . 

Tlie charts show that tor expeiimenl 1, the model qu^i^' accurately predicted a 
level oi reliahillty alter approximately 1/2 of the total uncovered faults were 
reported. The chart also shows that lor experiment 2, the model was still 
predicting a very high number ot errors to be still In tho software, when In 
tact a minimal set weie ever uncovered during the several years of operation for 
that system. 

More detailed discussions can he tound in Reterence I and II. 
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Method o logies (Charts \ 1 through 20) 

As was mentioned earlier, one ot the major objectives of the SKL has been to 
measure the effectiveness ot various software development methodologies. The 
SKL has utili/ed selected development approaches in different applications 
software tasks and then has analyzed the process and product to study the 
relative impact of the approach. A summary of some of the results of the 
experimentation process is presented liere. 

Use of An Independent Verification and Validation Team ( Cha r t 18) 

Many software managers, developers and organizations have advocated the usage of 
an independent IV/iV team during the software development process. The major 
advantage ot following such an approach, it is claimed, will be the improvement 
in software reliability, quality, visibility, but not necessarily an improvement 
in over a 1 1 sof t ware product i vi ty . 

In an attempt to evaluate the impact that the usage of an IV&V team may have on 
the SKL environment, ^ candidate projects were selected to utilize the 
methodology ot an IV&V. IVo of the projects were very typical flight dynamics 
systems, each containing over SO, 000 lines of code while tlie third was a smaller 
flight dynamics project comprised of about 10,000 lines of code. In addition to 
the IV&V approach being applied to the projects, the development teams utilized 
the commonly followed standards and approaches normally used by development 
efforts within the SKL environment. 

The projects lasted approximately 18 months, and the TV&V effort was active for 
the entile duration of the project. The size of the IV&V effort was about 18 
percent of the effort of each of the large development efforts. A series of 
measures was defined near the beginning of the experiment by the SKL. These 
measures would be used to determine whether or not the application of the IV&V 
approach was cost effective in the SKL environment. 

A summary of some of the measures is depicted in Chart 18. The results here 
ind i cate : 


o total cost of the project lncreased~as expected 

o productivity of the development teams (not counting the cost of 
IV&V) was among the lowest of any previous SKL monitored project. 

o rates of uncovering errors found earlier in the development cycle 

was better 

o cost rate to fix all discovered errors was no less than in any 
other SKL projects 

o reliability of the software (error rate during acceptance testing 
and during maintenance and operations) was no different than other SKL projects 
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The conclusion of the SEL, based on these 3 experiments, was that the IV&V 
methodology was not an effective approach in this SEL environment. 

(A more detailed description can be found in Reference 12). 

Effects of MPP on Software Development (Chart 19) 

In an attempt to determine if the utilization of Modern Programming Practices 
(MPP) has any impact (either favorable or unfavorable) on the development of 
software, a set of 10 fairly large (between 50,000 l.o.c. and 120,000 l.o.c.), 
and fairly similar projects (same development environment, same type of 
requirements, same time constraints) was closely examined. These projects all 
had been developed in the SEL environment where detailed information was 
extracted from the projects weekly and where each project had different level 
of MPP enforced during the development process. 

The MPP's ranged from various design approaches (such as POL, Design Walk 
Throughs , etc.) to code and test methodologies (such as structured code, code 
reading, etc.), to various Integration and system testing approaches. All of 
the possible MPP’s were rated and scaled as to the level to which the practice 
was followed for each project (the rating was done by the SEL researchers, not 
by the software developers). The only purpose of this exercise was to depict 
trends and not to prove that any one single practice was more effective by 
itself than any other. 

The level to which MPP's were utilized were plotted against productivity and 
against error rate. Chart 19 indicates that the application of the MPP has 
favorably affected productivity by about 15 percent for these experiments. The 
results of software reliability vs MPP is very questionable. The SEL Is still 
continuing analysis of additional data. The chart shown is obviously /ery 
Inconclusive . 

(More details of this effort can be found in Reference 13). 

Subjective Summary of Effective Practices (Chart 20) 

The previous chart indicated that productivity can be improved by an appreciable 
amount if certain, select practices are applied to the software development 
process. One obviously next would ask, which practices are the most effective? 
The SEL has been attempting to analyze the available data from the 40 
experiments it has conducted to answer this very question. As was F/cated 
earlier, the SEL feels that these types of experiments can only depict trends 
and cannot accurately isolate one practice as measurable on its own. Whether or 
not this can be done, or whether one should ever attempt it is questionable. 

Most software development methodologies represent an integrated set of practices 
that only are effective when they are applied in a combined, uniform fashion. 
Most practices do not make sense, or at least cannot be effective as a stand 
alone approach. 
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A summary ot ih.c trends that the SKL has discovered tor specific experiments 
conducted Is represented In Chart 20. This chart Is a combination of 
experimental results and subjective information from the experimenters and users 
and should only be viewed as depicting trends in various approaches. No 
numerical value of impact can realistically be assigned to the Individual 
practices tested. It seems that practices such as PDL, code reading and 
librarian have proved most beneficial while such techniques as automated flow 
charters, requirements languages and the axriomatic design approach have been 
unsuccessful in the SKL. 

Cost of D ata Collec tion ( Cha r t 21) 

The SKL has been in existence for about 7 years and has been collecting detailed 
software development data for over 6 years. Numerous experiments have been 
conducted in an attempt to understand and measure various methodologies for 
developing software. Tn support of these efforts, one of the most critical and 
difficult elements of the entire experimentation process is that of data 
collection . 

The data collection process is time consuming, frustrating, sometimes 
unrewarding, and most assurably Is expensive. Chart 21 shows the overhead cost 
that the SKI. has experienced over the past b years. To accurately collect data 
from the development tasks, the SKL finds that there is a 3 to 7 percent 
overhead price on the development effort. To process the data that has been 
collected (verification, encoding, data entry, storage, etc.), the SKL has spent 
approximately an additional 10 to 12 percent of the development effort. Finally, 
the SKL experiences Indicate that one can spend up to an additional 25 percent 
of the development effort to perform the detailed analysis of the data that has 
been collected. This Includes support before, during and after the experiments 
In defining the data to be collected, monitoring the development data and 
effort, formulating hypothesis and performing analysis of the completed 
experiments. The product of the analysis consists of papers, reports, and 
documents * 

(Detailed information on cost can be found in Reference 2). 

Summary ( Chart 22) 

In summary, the SKL has had much experience with the data collection process and 
with the experimentation process. Many of Its attempts have been rewarding and 
many have been fruitless, but the SKL feels attempts to assess approaches to 
software have to be conducted if we are ever to evolve to a more productive 
approach to developing software. 
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MEASURING 

SOFTWARE DEVELOPMENT 

TECHNOLOGY 

OR 

SHOULD PROGRAMMERS DO IT 

TOP DOWN ? 


3MP AG 0 1 


CHART I 
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SEL APPROACH TO SOFTWARE 
TECHNOLOGY ASSESSMENT 


SOFTWARE EXPERIMENTS IN PRODUCTION ENVIRONMENT: 

NASA APPLICATIONS 


• DEVELOP PROFILE OF ENVIRONMENT 
(SCREENING) 


• EXPERIMENT WITH PROPOSED 
TECHNOLOGIES (CONTROLLED) 


• MEASURE IMPACT AND/OR ASSESS 
TECHNOLOGIES 


- EXTRACT DETAILED DEVELOPMENT 
DATA 

- DETERMINE CHARACTERISTICS OF 
DEVELOPMENT PROCESS 

- APPLY VARIOUS TECHNOLOGIES 
(METHODS, MODELS, AND TOOLS) TO 
APPLICATIONS PROGRAMS 

- EXTRACT DETAILED DEVELOPMENT 
DATA 

- DEFINE MEASURES FOR EVALUATION 

- COMPARE EFFECTS OF USING OR NOT 
USING APPROACHES IN QUESTION 
(SIMILAR PROJECTS) 

- DETERMINE EFFECTIVENESS OF 
TECHNOLOGIES IN QUESTION (WHICH 
ONES HELP AND BY HOW MUCH) 


CHART 2 


m P M ocn 
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SOFTWARE ENVIRONMENT 


DEVELOPMENT LANGUAGE 
SOFTWARE TYPE 

SIZE 

DEVELOPMENT TIME 

STAFFING 

DEVELOPMENT SYSTEM . . . 


FORTRAN (15% MACROS) 

SCIENTIFIC, GROUND- 
BASED INTERACTIVE, 
NEAR-REAL-TIME 

TYPICALLY'>^60,000 SLOC 
(2,000 TO 110,000) 

16 TO 24 MONTHS (START 
DESIGN TO START 
OPERATIONS) 

6 TO 14 PERSONS 

IBM S/360 (PRIMARILY) 
VAX-11 /780 
PDP-11 /TO 


3M^AiG-0*l 


CHART 3 
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EXPERIMENTS WITHIN THE SEL 
1977 THROUGH 1982 
BASIS FOR SUMMARY INFORMATION 

AND CONCLUSIONS 


LABORATORY EXPERIMENTS 46 PROJECTS 

INFORMATION MONITORED 1.8 MILLION SLOC 

PROGRAMMERS/MANAGERS 

REPRESENTED 150 PEOPLE 

DATA EXTRACTED 20,000 FORMS 

METHODOLOGIES APPLIED 200 QUALIFYING PARAM- 

ETERS AND VARIOUS 
MODELS, TOOLS, AND 
METHODOLOGIES 


384-PAOai 


CHART 4 
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AREAS OF DISCUSSION 


• PROFILES 

• MODELS 

• METHODOLOGIES 


33«^Ae42*l 


CHART 5 



PROFILES 


33«-PAG-12*> 


CHART 6 
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WHERE DO 

PROGRAMMERS SPEND THEIR TIME? 


DATE DEPENDENT PROGRAMMER REPORTING 



3M4>A6-4ri 


CHART 7 
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PROFILES OF DEVELOPMENT RESOURCES 



DESIGN CODE AND SYSTEM ACCEPTANCE 

UNIT TESTING TESTING TESTING 


PROFILE 

PRODUCTIVITY 

ISLOC/HOUR) 

TIME ► 

REUABILITY 
(ERRORS/K SLOC) 

• RELATIONSHIP BETWEEN 


RAYLEIGH CURVE 


PROFILE AND 
PRODUCTIVITV 


4.4 -4.6 
2.T-4.7 

UP TO 2 
UP TO 2 

• NO RELATIONSHIP 
BETWEEN PROHLE AND 


2.7- 2.9 

UP TO 2 

RELIABILITY 
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ARE LARGE PROGRAMS 
HARDER TO BUILD THAN SMALL ONES? 




^ ^ 
o 0^2 

•$;a? 
ss =5 



DEVELOPED LINES OF CODE (THOUSANDS) 
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PRODUCTIVITY VARIATION (SLOC/HOUR)'' 


BY PROJECT 
(ALL CHARGES) 




BY PERSON 
(PROGRAMMER ONLY) 



PEOPLE ARE THE MOST IMPORTANT METHODOLOGY 
^A LARGE PROJECT IS GREATER THAN 20K SLOC. 
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1 ' 

N ASA CSK' 


DEVELOPED SLOC 


DEV SLOC NEW 

I I 

0 10 20 30 


40 50 60 70 80 90 

NEW CODE i^o) 


100 

3M-MG (2a*l 
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ERROR CHARACTERISTICS 

(MEASURED DURING IMPLEMENTATION) 

TYPES OF ERRORS EFFORT TO CORRECT 




SAMPLE OF 1381 REPORTS 

• MOST ERRORS ARE EASY TO CORRECT 

• SEVERAL-COMPONENT ERRORS ARE LESS THAN EXPECTED 

• REQUIREMENTS ERRORS ARE LESS THAN EXPECTED 

J]1 PAO lf•■l 
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MODELS 

33M»A&Ob*t 
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RELIABILITY 

(ERRORS PER LINE OF CODE) 


SOFTWARE MEASURES IN THE SEL 



I.- , . i r, iw **** *•« '‘•w? 

McCABE COMPLEXITY UNkS OF CODE 

CORRELATIONS 



TOTAL 

LINES 

EXECUTABLE 

LINES 

McCAPE 

COMPLEXITY 

HALSTEAD 

LENGTH 

HALSTEAD LENGTH 

0.85 

0.91 

0.91 

1.00 

McCABE COMPLEXITY 

0.81 

0.87 

1.00 


EXECUTABLE LINES 

0.84 

1.80 



TOTAL UNES 

1.00 





SAMPLE OF 688 MODULES 
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ORIGINAL PAGE IS 
OF POOR QUALITY 
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COMPARISON OF COST MODELS 


ACTUAL PERCENTAGE OF ERROR IN PREDICTION 

EFFORT 


PROJECT 

(MM) 

DOTY 

PRICE S3 

TECOLOTE 

SEL 

COCOMO 

1 

79 

+ 65 

+ 8 

-4 

-6 

— 

2 

96 

+ 30 

+ 6 

-25 

-22 

+ 1 

3 

40 

+ 65 

+ 6 

-8 

+ 93 

— 

5 

98 

+ 74 

0 

+ 3 

-2 

+ 2 

6 

116 

+ 123 

^36 

+ 35 

-3 

— 

7 

91 

+ 52 

14 

-12 

-14 

— 

8 

99 

+ 127 

7 

+ 36 

+ 14 

+ 53 

9 

106 


- 

■ 

-24 

+ 16 


SOMETIMES, SOME MODELS WORK WELL 


334-PAG-(2b*l 
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PREDICTING RELIABILITY 

(MUSA MAXIMUM LIKELIHOOD METHOD) 


^ Z H 
> ■ 

c W 2 
C) 

CJ fis 

^ C/5 q 

n 



WE DONT KNOW ENOUGH ABOUT REUABIUTY MODELS 
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METHODOLOGIES 


33«-PA6-ak*t 
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ERRORS/K ExLOC MAN-MONIHS/K SLOC 


A LOOK AT IV&V METHODOLOGY 

(BASED ON RESULTS FROM 3 EXPERIMENTS) 


2.^2 

^ O C3 
C» 2 

o 






• IF YOU MULTIPLY ERRORS FOUND EARLY BY A LATENCY 
FACTOR, IV&V LOOKS GOOD 

• IF YOU EXAMINE AU MEASURES, IVGV LOOKS BAD 

mMMin 
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DEVELOPED LINES 
OF CODE PER HOUR 


EFFECTS OF MPP 

ON SEL SOFTWARE DEVELOPMENT 


PRODUCTIVITY 



500 1000 1500 

INDEX OF MODERN 


PROGRAMMING PRACTICES 


ERROR RATE 



1 • 

500 1000 1500 

INDEX OF MODERN 


PROGRAMMING PRACTICES 


• PRODUCTIVITY IS ABOUT 15 PERCENT HIGHER 


• RELIABILITY IS HIGHLY VARIABLE 
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OVERHEAD COST 


WHAT HAS BEEN SUCCESSFUL IN OUR ENVIRONMENT? 
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COST OF DATA COLLECTION 

(AS A PERCENTAGE OF TASKS BEING MEASURED) 


OVERHEAD TO TASKS (EXPERIMENTS) 

• FORMS 

• MEETINGS 

• TRAINING 

• INTERVIEWS 

• COST OF USING TOOLS 

DATA PROCESSING 

• COLLECTING /VALIDATING FORMS 

• ARCHIVING /ENTERING DATA 

• DATA MANAGEMENT AND REPORTING 

ANALYSIS OF INFORMATION 

• DESIGNING EXPERIMENTS 

• EVALUATING EXPERIMENTS 

• DEFINING ANALYSIS TOOLS 


SEL 

EXPERIENCES 

3-7% 


10 - 12 % 
UP TO 25% 
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SUMMARY 


• DATA COLLECTION IS EXPENSIVE - BUT 
VERY, VERY IMPORTANT 

• WE MUST UNDERSTAND WHERE WE ARE 
BEFORE HEADING SOMEWHERE ELSE 

• EXPERIMENTATION WILL PAY FOR ITSELF (TRY 
SOMETHING NEW) 

• MPP CAN FAVORABLY IMPACT PRODUCTIVITY 
AND RELIABILITY 

• SOME METHODOLOGIES BUY YOU NOTHING 
(OR EVEN WORSE) 

• MODELS MUST BE UTILIZED WITH GREAT 
CARE 
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